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(54) Image processing apparatus, communication device and system thereof 



(57) A video communication system provided with a 
plurality of video communication devices connected to 
a network comprising a circuit for selecting and display- 
ing video data received from the above-described video 



communication device and video data received from a 
video storage apparatus, and the video storage appa- 
ratus comprising a video data storage means and a 
means for transmitting required video data to the video 
communication device in case a request for reading is 
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SUMMARY OF THE^ff NTION 



Description 

BACKGROUND OF THE INVENTION 

Field of the Invention 

The present invention relates to an image process- 
ing apparatus, a communication device and a system 
thereof, particularly toa video communication device re- 
quired for displaying image data and a system thereof. 

Related Background Art 

If digital video data Is reproduced as a dynamic Im- 
age by a computer, in the prior art a digital video data 
file Is successively read and displayed on the screen of 
a monitor after a file is decompressed in case it is com- 
pressed. Therefore, normally only an image at a time in 
a video data file can be seen. According to the prior art, 
video data consisting of different files cannot be repro- 
duced and displayed continuously 

For instance, if the past video data stored In a video 
sender is desired to be read when the current Image pro- 
vided by a video camera connected to a computer in a 
network Is communicated to another computer via the 
network and is recorded in the video server in the net- 
work as a file divided every given time, repetitive work 
in which another file Is read after reading a file is com- 
pleted is required. 

To know how an Image changes after given time 
elapses when a reproduced image is seen, an operation 
for reading, reproducing and displaying another file is 
required if the image exists in another file, and manual 
setting of time is required even if the Image exists in the 
same file. 

Furthermore, in a video conference system and a 
supen/isory control system for monitoring a remote sta- 
tus, an image provided by a video camera is transmitted 
and displayed via a network. 

In the meantime, a system in which an image pro- 
vided by a video camera is digitized, compressed ac- 
cording to an image coding scheme such as Moving Pic- 
ture Experts Group (MPEG) and Joint Photographic 
Coding Experts Group (Motion JPEG) if necessary and 
stored in an image database exists. 

However, if the past Image of the current one pro- 
vided by a camera is retrieved because a system for 
transmitting and displaying a remote image and a sys- 
tem retrieving and displaying an image stored in an im- 
age database are separate, both systems must be ac- 
tivated individually. 

Also, as both systems are separate, an image can- 
not be retrieved and displayed continuously from the 
past to the present and the operations are also incon- 
sistent. 



A concern of the present invention is to provide an 
Image processing apparatus, a video communication 
5 device and a system thereof wherein all or each of the 
above-described problems are/Is solved. 

Another concern of the present invention is to pro- 
vide an image processing process which enables read- 
ing video data stored in a video storage apparatus with 
10 complete control, a video communication device and a 
system thereof. 

Another concern of the present invention Is to pro- 
vide an image processing process which enables read- 
ing video data stored in a video storage apparatus via 
15 a communication line with complete control, a video 
communication device and a system thereof. 

In accordance with the present invention there is 
provided image processing apparatus provided with first 
means for supplying an input image to first display 
20 means to achieve real-time display and second means 
for supplying the Image prior by predetermined time of 
the input image to second display means to achieve re- 
al-time display. 

Further, another concern of the present invention is 
25 to provide an image processing apparatus, a communi- 
cation device or a system thereof provided with a new 
function. 

The other features and characteristics of the 
present Invention will be disclosed by description of em- 
30 bodiments and the drawings described below 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a schematic block diagram according to an 
35 embodiment of the present invention; 

Fig. 2 shows access to a file in the current mode; 
Fig. 3 Is a flowchart of displaying video data in the 
current mode; 

Fig. 4 shows access to a file in the past mode; 
^0 Fig. 5 is a flowchart of displaying video data in the 
past mode; 

Fig. 6 shows a graphical user interface in the current 
mode; 

Fig. 7 shows a graphical user interface in the past 
45 mode; 

Fig. 8 is a block diagram of a video communication 
system according to a second embodiment of a 
present invention; 

Fig. 9 is a block diagram showing a video commu- 

50 nication device according to a second embodiment 
of the present invention; 

Fig. 10 is a block diagram showing a video storage 
apparatus shown in Fig. 8 according to a second 
embodiment of the present invention; 
55 Fig. 11 Is a flowchart showing operation in video 
transmission processing by a video communication 
device shown in Fig. 9 according to a second em- 
bodiment of the present invention; 
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Fig. 1 2 is a flowchart showingOperation in video re- 
ceiving processing by a video communication de- 
vice according to a second embodiment of the 
present invention; 

Fig. 13 is a flowchart showing operation in video dis- s 
play processing by a video communication device 
according to a second embodiment of the present 
invention; 

Fig. 1 4 is a flowchart showing operation in video re- 
trieval processing by a video storage apparatus ac- io 
cording to a second embodiment of the present in- 
vention; 

Fig. 15 is a flowchart showing operation in video 
storage processing by a video storage apparatus 
according to a second embodiment of the present is 
invention; 

Fig. 1 6 shows an example of an entire video display 
screen by a video communication device according 
to a second embodiment of the present invention; 
Fig. 1 7 shows an example of an individual video dis- 20 
play screen by a video communication device ac- 
cording to a second embodiment of the present in- 
vention; and 

Fig. 18 shows an example of a communication data 
format according to a second embodiment of the 25 
present invention. 

DESCRIPTION OF THE PREFERRED 
EMBODII\/IENTS 



Ir 



Referring to the drawings, preferred embodiments 
according to the present invention will be described in 
detail below. 

<First Embodiment> 



30 



35 



Fig. 1 is a schematic block diagram showing an em- 
bodiment according to the present invention. A video 
server 12 and a plurality of computers 14-1, 14-2 and 
1 4-3 are connected to a network 1 0 and video cameras 40 
16-1 to 16-3 as Image input means are connected to 
each computer 14-1 to 14-3. Each computer 14-1 to 
14-3 is basically constituted similarly and is provided 
with CPU 20 for controlling the entire computer, memory 
22 which is the main storage of CPU 20. a hard disk 24 ^ 
as a secondary memory, a bit map display 26, a mouse 
28 as a pointing device, a camera control circuit 30 for 
controlling a pan, a tilt, zooming in and out and a focus 
of the connected video cameras 1 6 and a video capture 
circuit 32 for fetching video output from the connected so 
video cameras 14 in the corresponding computers 14. 

This embodiment is provided with the current mode 
in which video output from a video camera 1 6 connected 
directly to a computer or to the other computer is repro- 
duced and displayed at a real time and the past mode ss 
in which a video data file stored in a video server 1 2 is 
reproduced and displayed. A video server according to 
this embodiment is constituted by a well-known system 



called Redundant Arr^Wi Inexpensive Disc (RAID) and 
enables concurrent writing and reading of dynamic im- 
age data. 

First, the current mode will be described below. Fig. 
2 shows access to a file in the current mode and Fig. 3 
shows a flowchart of operation in the current mode. 

Referring to Fig. 2, files 1 , 2, etc. are a series of files 
consisting of digital video data provided by the same one 
video camera 16 and divided every given time TO and 
numbers are allocated to them in the order of genera- 
tion. Tb shows time in which video data is not read for 
processing for dividing into files. It is naturally preferable 
that Tb is as short as possible and Tb can be substan- 
tially removed by securing sufficient capacity of a buffer, 
"t" shows the current time, tp shows the time of the past 
video (frame) displayed together with the current video 
(frame) and Ts shows the difference of time between the 
current and past video concurrently displayed, "tp" Is 
equal to a value obtained by subtracting Ts from "t". In 
this embodiment, a value obtained by adding TO and Tb 
is set to a smaller value than Ts. That is, the past video 
concurrently displayed is stored In a file prior by one or 
more depending upon the ratio of Ts to TO. 

When the current mode is activated for instance by 
operating a mouse shown in Fig. 1, Ts is first set (S1) 
and current time data is set to a variable t showing the 
current time (S2). An activated video browser starts dis- 
playing an image provided by a specified video camera 
16 in the main window of a bit map display 26 and con- 
tinues to display an image in the main window at a real 
time until termination is specified (S3, S7). 

When time specified for Ts elapses (S4) since dis- 
playing a video signal provided by a video camera 1 6 in 
the main window is started, displaying a frame at the 
time tp prior by Ts to the current time t in a sub window 
is started (S5), a variable t is updated (S2) and elapse 
by Ts is again awaited. As it is warranted that a frame 
at the time tp is recorded as a file without fall according 
to a condition, "TO + Tb < TS", the frame may be read 
from the file. However, exceptionally, the time tp may be 
just in the range of Tb and in this case, a frame at the 
nearest time prior to the time tp of the files in which video 
is recorded may be selected. 

If Ts must be changed on the way (S6), operation 
is restarted from S1 at that time. 

As described above, as the current video and the 
video prior by time defined as Ts are concurrently dis- 
played according to the current mode in this embodi- 
ment as shown in Fig. 3, change of time can be readily 
checked and when the current video is to be taken notice 
of, it can be checked by checking the past video in detail. 
Such a function is very useful for a variety of monitoring 
camera systems. 

Next, the past mode will be described. In the past 
mode, the main reproduced video is displayed in the 
main window and the past and future Images of which 
difference in time Is equivalent to 2Ts with the main re- 
produced video in the middle of the time 2Ts are dis- 
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played in an individual subwindowSfa static image. The 
past and future static Images are respectively updated 
to such images after 2Ts elapses at the appropriate tim- 
ing. Fig. 4 shows access In the past mode and Fig. 5 
shows a flowchart In the past mode. s 

Referring to Fig. 4, "f" shows the time in which a 
frame Is displayed In the main window or a frame pointer 
and it is equivalent to the center in time between the 
past and future Images displayed in the subwindow or 
a frame pointer, "tp" shows the time prior by Ts to tc or io 
a frame pointer and "tf " shows the time later by Ts than 
tc or a frame pointer. That is, tc = tp + Ts = tf - Ts. "tc" 
shows the time for reproduction to be started and deter- 
mined by a viewer. 

If video is displayed in the past mode, a viewer first is 
sets the time tc of a frame of which reproduction is to be 
started and Ts which defines images prior and later how 
In time with tc In the center are displayed in a subwindow 
(S11 ). "tc" is set to "f" (S12). "tp" and "tf" are calculated 
based upon tc and Ts (SI 3). The frames at the time tp 20 
and at the time tf are read of a series of video files and 
are displayed in respective subwindows (S14). 

Next, a frame at the time t' of which initial value is 
tc is fetched of a series of video files similarly and is 
displayed in the main window (S15). Each frame is se- 2S 
rlally reproduced and displayed (S15) adding the time 
dt between frames to the time V (S1 9) until an instruction 
for termination is Input (S18). 

If the time f Is equal to tf in the meanwhile (SI 6), It 
Is added by 2Ts to tc (SI 7), tp and tf of the new tc are 30 
again calculated (S13) and Images to be displayed in 
subwindows are updated (S14). 

Though action In case Ts is changed on the way of 
reproduction Is omitted in Fig. 5, in that case operation 
may be restarted from a step S1 1 . 55 

As described above, the past dynamic Image be- 
tween tp and tf based upon tc and Ts arbitrarily set can 
be displayed In the main window and the frames at the 
time tp and at the time tf can be displayed In a respective 
subwindow. As not a dynamic image but a static Image 40 
is displayed in a subwindow, only one circuit for repro- 
ducing a dynamic Image for the main window is required 
and a dynamic image can be provided at a low cost. As 
a static image displayed in a subwindow Is updated eve- 
ry 2Ts, It can be also considered a so-called low-speed 45 
dynamic Image and a system efficient for the cost can 
be provided. 

In this embodiment, tc and Ts may be input to CPU 
20 using an input device such as a mouse shown in Fig. 
1. 50 

Figs. 6 and 7 show examples of an operator panel 
and a display screen of a bit map display 26. Fig. 6 
shows the screen when the current mode Is selected 
and Fig. 7 shows the screen when the past mode is se- 
lected. In both cases, a physical arrangement such as S5 
the position of a button is the same. 

Referring to Figs. 6 and 7, a reference number 40 
designates the main window for displaying video, a ref- 
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erence number 42 de^nates a dialog for setting tc in 
case video Is displayed in the main window 40 in the 
past mode, and a reference number 44 designates a di- 
alog for setting Ts. A reference number 46 designates 
a subwindow for displaying a frame at the time tp and a 
reference number 48 designates a subwindow for set- 
ting a frame at the time tf. 

A reference number 50 designates a time slider bar 
for specifying the time between tp and tf of a frame to 
be displayed in the main window 40 and displaying the 
frame and the time slider bar Is available only when the 
past mode is selected. A reference number 52 desig- 
nates a time gauge between tp and tf. 

Reference numbers 54, 56, 58, 60, 62 and 64 re- 
spectively designate a reproduction control button for 
controlling video (a dynamic image) between tp and tf 
displayed in the main window 40. The button 54 is used 
for specifying processing for subtracting 2Ts from a val- 
ue of tc, the button 56 is used for specifying processing 
for rewinding, the button 58 is used for specifying 
processing for reproduction, the button 60 is used for 
specifying processing for a pause, the button 62 is used 
for specifying processing for a rapid traverse and the 
button 64 is used for specifying processing for adding 
2Ts to a value of tc. 

A reference number 66 designates a current mode 
button for setting the current mode and a reference 
number 68 designates a past mode button for setting 
the past mode. 

A reference number 70 designates a continuous re- 
production mode button and a reference number 72 
designates an interval reproduction mode button. Con- 
tinuous reproduction is enabled by pressing the contin- 
uous reproduction mode button 70. In this embodiment, 
a dynamic image can be reproduced only in the interval 
between tp and tf basically defined based upon tc and 
Ts, however, if continuous reproduction is enabled, tc is 
automatically updated when a frame at the time tf is re- 
produced and reproduction in a new interval is contin- 
ued. As a result, in the continuous reproduction mode, 
continuous reproduction without limitation on an Interval 
between tp and tf Is enabled. Continuous reproduction 
can be invalidated by pressing the interval reproduction 
mode button 72. 

Reference numbers 74 to 86 respectively designate 
a button for operating a camera and a reference number 
88 designates a panel for operating a camera. The but- 
ton 74 is used for specifying a pan to the left side of a 
camera, the button 76 Is used for specifying a pan to the 
right side, the button 78 Is used for specifying an upward 
tilt, the button 80 is used for specifying a downward tilt 
and the button 82 is used for specifying an operation for 
directing a camera to the center. The button 84 is used 
for specifying operation for Increasing the magnifying 
power of the lens, that is, movement to a telephoto lens 
and the button 86 Is used for specifying operation for 
reducing the magnifying power of the lens, that Is, move- 
ment to a wide angle lens. The operator panel 88 shows 
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the direction and the magnifyin^^wer of the camera 
by the position and the size of a four-sided figure in a 
frame and also displays the current status. 

The dialog for setting tc 42, the time slider bar 50. 
the time gauge 52, the buttons 54 to 64, the current s 
mode button 66, the continuous reproduction mode but- 
ton 70 and the interval reproduction mode button 72 are 
enabled only in the past mode. In the meantime, the past 
mode button 68, the buttons for operating a camera 74 
to 86 and the panel for operating a camera 88 are ena- io 
bled only in the current mode. Disabled objects are dis- 
played in gray out or are not displayed. 

In the current mode, a pan, a tilt and zooming of a 
specified camera can be operated by pressing an ap- 
propriate button of the buttons for operating a camera ^5 
74 to 86 or operating the panel for operating a camera 
88. In a window display system, when a mouse button 
is clicked at an arbitrary point in a window, the coordi- 
nates at that point can be fetched. When an arbitrary 
point on the screen of the main window 40 is clicked, a 20 
camera can be panned or tilted so that the clicked point 
is in the center using such a function and further, a cam- 
era can zoom in or out so that an appropriate range can 
be photographed. For instance, when an object or a per- 
son to be taken notice of is clicked, a camera can be 25 
controlled so that the camera Is directed in the direction. 

In the current mode, a frame at the time tp is dis- 
played in the left subwindow 46. This provides plainness 
sensible by intuition. CPU 20 controls a bit map display 
so that a message or a mark showing the system is in 30 
the current mode is displayed in the right subwindow 48. 

In the past mode, reproduction can be controlled by 
pressing an appropriate button of the reproduction con- 
trol buttons 54 to 64 which function as reproduction con- 
trol buttons provided to a video tape recorder. The but- 35 
tons 54 and 64 are provided with characteristics peculiar 
to this embodiment and are used for changing an Inter- 
val In time specified by the time gauge 52. In Fig. 7, an 
example in which Ts = 5 (min.), tc = three thirty in the 
afternoon, tp = three twenty-five in the afternoon and tf 40 
= three thirty-five in the afternoon is shown, however, 
when the button 54 is clicked, tc is changed to three 
twenty in the afternoon, inevitably, tp is changed to three 
fifteen in the afternoon and tf is changed to three twenty- 
five In the afternoon. When the button 64 is clicked, tc ^5 
is changed to three forty In the aftemoon. That is, video 
like that by a movie film is enabled on the three screens 
of the subwindow 46, the main window 40 and the sub- 
window 48 by clicking the buttons 54 and 64 many times. 

Further, as the Interval in time of a frame displayed 50 
in the main window 40, the subwindows 46 and 48 can 
be changed in the dialog 44, a viewer can view at an 
arbitrary interval In time. This function is very effective 
in case the system is used for instance as a crime pre- 
vention camera system or a nrranitoring camera system, ss 
In case anything is always photographed by a camera, 
in the case of a video tape, any video recorded on the 
video tape must be reproduced and displayed by a rapid 
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traverse when changWf video is searched, however, 
according to this embodiment, the video can be ac- 
cessed without continuation in time and the time re- 
quired for finding a necessary point can be reduced sub- 
stantially. 

As video at three times can be concurrently seen, 
change can be readily found. For instance, the system 
is also effective for a person who is late for a video con- 
ference to understand the outline of the entire confer- 
ence for a short time. Continuous reproduction is ena- 
bled by pressing the continuous reproduction mode but- 
ton 70. Basically, only video between tp and tf is dis- 
played, however, if continuous reproduction is validated; 
tc is automatically updated when a frame at the time tf 
Is reproduced and reproduction at a new inten/al is con- 
tinued. As a result, in the continuous reproduction mode, 
video is continuously reproduced without limitation on 
the interval between tp and tf. Continuous reproduction 
can be invalidated by pressing the interval reproduction 
mode button 72. 

A series of video can be effectively viewed and the 
operations can be also extremely easy by adopting a 
graphical user interface as shown In this embodiment. 

As understandable readily by the description 
above, as according to this embodiment a series of vid- 
eo files divided every given time can be continuously 
handled, a viewer can access to video files for a long 
time without being conscious of each file. By this, really 
required sections can be readily fetched from massive 
video data. 

Also, as the current video or video at the required 
time and the past image or the past and future images 
are concurrently displayed, change in video can be com- 
pared on the screens. The change in video Is readily 
sensible and a desired point can be readily found. As 
the past and future images are displayed as a static im- 
age, a circuit for display can be simple and a very effec- 
tive video display system at a low cost can be provided. 

<Second Embodiment> 

Referring to the drawings, a second embodiment 
according to the present invention will be described be- 
low. 

Fig. 8 is a block diagram showing a video commu- 
nication system according to this embodiment. 

Referring to Fig. 8, reference numbers 101 to 106 
respectively designate a video camera, reference num- 
bers 111 to 116 respectively designate a video commu- 
nication device and a reference number 1 20 designates 
a video storage apparatus. The video camera are con- 
nected respectively to video communication devices 111 
to 116. 

A reference number 130 designates a network for 
connecting the video communication devices 1 1 1 to 1 1 6 
and the video storage apparatus 120, and a local area 
network (LAN) and a wide area network (WAN) such as 
the telephone network, ISDN and the common carrier 
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leased line are included. 

Fig. 9 is a block diagram showing video communi- 
cation devices 1 1 1 to 1 1 6 according to this embodiment. 

Referring to Fig. 9, a reference number 101 desig- 
nates a video camera, a reference number 100 desig- s 
nates an intemal bus and a reference number 121 des- 
ignates a camera control apparatus for controlling a pan, 
a tilt and zooming of the camera 101. A reference 
number 1 22 designates a video input control apparatus 
for fetching in the video communication device after a io 
video signal fetched from a video camera is converted 
from analog to digital, a reference number 123 desig- 
nates a video encoding/decoding apparatus for com- 
pressing or decompressing digital video information, 
and a reference number 1 24 designates a main control 15 
apparatus for controlling all the video communication 
devices 121 A reference number 125 designates a pri- 
mary storage for storing a variety of processing proce- 
dures (programs), communication data and control in- 
formation, and a reference number 126 designates a 20 
secondary memory which enables high-speed writ- 
ing/reading of data. A reference number 127 designates 
an operation apparatus for a pointing device (mouse), 
a reference number 1 28 designates a display apparatus 
such as a bit map display, and a reference number 1 29 25 
designates a communication control apparatus for con- 
trolling communication with a network 130. 

Fig. 10 is a block diagram showing a video storage 
apparatus 120 according to this embodiment. 

Referring to Fig. 10, a reference number 200 des- 30 
Ignates an Internal bus, a reference number 203 desig- 
nates a video encoding/decoding apparatus (CODEC) 
for compressing or decompressing digital video infor- 
mation, and a reference number 204 designates a main 
control apparatus for controlling the entire video storage 3S 
apparatus 120. A reference number 205 designates a 
primary storage for storing a variety of processing pro- 
cedures (programs), communication data and control 
Information, a reference number 206 designates a sec- 
ondary memory which enables high-speed writing/read- 40 
Ing of data, anda reference number 207 designates an 
operation apparatus for a pointing device (mouse). A 
reference number 208 designates a display apparatus 
such as a bit map display and a reference number 209 
designates a communication control apparatus for con- 45 
trolling communication with a network 130. 

Figs. 11 to 13 are flowcharts showing operation of 
the video communication devices 111 to 116 according 
to this embodiment. 

Figs. 14 and 15 are flowcharts showing operation so 
of the video storage 1 20 apparatus according to this em- 
bodiment. 

Fig. 16 shows an example of the screen displaying 
the entire display apparatus 118 of the video communi- 
cation devices 1 1 1 to 1 1 6 according to this embodiment, 
Each video communication device 1 1 1 to 1 1 6 according 
to this embodiment is a terminal used for, for instance a 
video conference or a visual telephone and video win- 
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dows 91 0 to 960 corr^^nding to each participant of a 
video conference are shown in Fig. 16. 

Referring to Fig. 16, reference numbers 910 to 960 
designate video windows for receiving and displaying 
video transmitted by the video communication devices 
111 to 116 to which video cameras 101 to 106 are con- 
nected, and reference numbers 911 to 961 respectively 
designate a status for displaying the name of a calling 
party and the time in which video is Input corresponding 
to each video window 910 to 960. 

Fig. 1 6 shows an example of the screen for display- 
ing individual video in a display section 118 of the video 
communication devices 1 1 1 to 1 1 6 according to this em- 
bodiment. 

Referring to Fig. 1 7, a reference number 1010 des- 
ignates a video window for receiving and displaying vid- 
eo transmitted by the specified one of the video com- 
munication devices 111 to 116. Reference numbers 
1020 and 1030 designate camera control panels for 
controlling the video cameras 101 to 106 via a camera 
control apparatus 111 and they issue an instruction to 
the camera control apparatus 111 by clicking a variety 
of graphical user interfaces (GUI) via a mouse to enable 
a pan, a tilt (1020) and zooming (1030) of a camera. 

A reference number 1050 designates a time spec- 
ification panel for Inputting a time and when a time is 
input to a time input field 1 051 and a setting button 1 052 
is clicked, dynamic Image data stored in the video stor- 
age apparatus 120 is accessed to retrieve data corre- 
sponding to the specified time. 

A reference number 1040 designates a switch for 
switching video from the video communication devices 
111 to 116 and reading dynamic image data stored in 
the video storage apparatus 120, and when the current 
mode button 1041 is clicked, video transmitted by the 
video communication devices 111 to 116 is displayed in 
a dynamic image display window 1010. When the past 
mode button 1 042 is clicked, dynamic image data stored 
In the videostorage apparatus 1 20 Is read and displayed 
in the dynamic Image display window 1010. When a time 
is input to a time specification panel 1050 and a setting 
button 1 052 is clicked, dynamic Image data stored in the 
video storage apparatus 20 is retrieved and displayed. 

Fig. 18 shows an example of a data format used for 
the video communication system according to this em- 
bodiment and the format is constituted by a header com- 
prising fields 141 to 145 and data 146. 

The header 141 to 145 comprises a packet type 
field 141 for specifying a variety of communication com- 
mands, a calling party's name field 142 for writing the 
name of a calling party, a called party's name field 143 
for writing the name of a called party, a media type field 
144 for writing a media type of a character, a static im- 
age, voice and a dynamic image and the other field 1 45 
for writing the other additional information. All media 
which may be handled as digital data may be written to 
the data field 146. 

The details of video transmission processing in the 
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video communication device l^^onstituted as de- 
scribed above according to this embodiment will be de- 
scribed below based upon a flowchart shown in Fig. 11 . 
However, as described above, the video communication 
device 1 1 1 executes the following processing according s 
to a mail processing program stored in a primary storage 
125 under control by a main control apparatus 124. 

First, in a step S401 , generation of an event is await- 
ed. When an event is generated, processing proceeds 
to a step S402 and in the step, it is checked whether the io 
type of the event is an instruction for operating a camera 
or not. If the type of the event is an Instruction for oper- 
ating a camera, processing proceeds to a step S403 and 
in the step, video cameras 1 01 to 1 04 are controlled via 
a camera control apparatus 111. 15 

A command for operating a camera comprises tilt 
(6), pan (0), zoom (z) ... (1) and according to these com- 
mands, a tilt, a pan and zooming of a camera are con- 
trolled. 

These commands are allocated to camera control 20 
panels 1020 and 1030 shown in Fig. 17. While an up- 
ward/downward button in the camera control panel 1020 
is pressed, a tilt (8) command is called increasing/de- 
creasing an angle 8. Similarly, while a left-hand/right- 
hand button in the camera control panel 1020 is 2S 
pressed, a pan ((])) command is called increasing/de- 
creasing an angle When a scroll bar in the camera 
control panel 1030 Is operated, a zoom (z) command is 
called increasing/decreasing the magnifying power z of 
zooming. 30 

However, operations of a camera may be instructed 
by the local video communication device or by the re- 
mote video communication device. 

In the step S402, if the type of an event is not an 
instruction for operating a camera, processing proceeds 35 
to a step S404 and in the step, it is checked whether 
there is an input to a camera or not. if the video camera 
101 connected to the video communication device 111 
is normally operated, processing proceeds to the step 
S404 and in the step, a video signal fetched by a video ^0 
input control apparatus 11 2 is converted from analog to 
digital. 

Next, processing proceeds to a step S406 and in 
the step, digital video data is compressed by video CO- 
DEC 1 1 3, processing proceeds to a step S407 and after ^5 
a communication data format described later is gener- 
ated, the compressed digital video data is transmitted 
to the network 1 30 via a communication control appa- 
ratus 119. In this case, communication data is required 
to comply with the format shown in Fig. 1 8, a code show- 50 
ing "transmission of video" is written to the packet type 
field 141 , the name of the video communication device 
111 is written to the calling party's name filed 142, the 
name of the other video communication devices 11 2 to 
11 6 or the video storage apparatus 1 20 is written to the ss 
called party's name field 1 43, a code showing "a dynam- 
ic image" is written to the media type field 144, the time 
in which video is fetched is written to the other field 145 
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and digitized video dalRs written to the data field 1 46. 

For the above-described name of a called party, "a 
broadcast address" by which any device in a network 
can be specified may be used. Further, if necessary, "a 
multicast address" by which a specific plural devices can 
be specified or "a unicast address" by which a specific 
device can be specified may be used. 

By the above-described processing, video informa- 
tion is transmitted at a real time form each video com- 
munication device 111 to 116 to the network 1 30. 

Next, the details of video receiving processing by a 
video communication device 115 according to this em- 
bodiment will be described below based upon a flow- 
chart shown in Fig. 12. 

First, in a step S501 , generation of an event is await- 
ed. When an event is generated, processing proceeds 
to a step S502 and in the step, it is checked whether the 
type of the event is reception of video or not. 

If communication data shown in Fig. 18 is received 
from the network 1 30 via the communication control ap- 
paratus 129 shown in Fig. 9, the packet type field 141 
is checked, if a code showing "transmission of video" is 
identified, processing proceeds to a step S503, in the 
step, the entire video display processing shown in an 
example of the screen in Fig. 16 is performed and 
processing is returned to the step S501. If the name in 
the called party's name field 1 04 does not show the local 
video communk;ation device, the received data is dis- 
carded. 

If the type of an event is not reception of video in 
the step S502, processing proceeds to a step S504 and 
in the step, it is checked whether a specific calling party 
of video is selected or not. Operations for selecting a 
specific calling party of video is performed by clicking 
an arbitrary one video window of video windows 910 to 
960 shown in Fig. 16. If a specific calling party of video, 
for instance one of the windows 910 to 960 shown in 
Fig. 1 6 is selected in the step S504 by the operation ap- 
paratus 117 shown in Fig. 9, processing proceeds to a 
step S505 and in the step, video retrieval processing de- 
scribed later is performed and processing is returned to 
the step S501 . 

Next, referring to a flowchart shown in Fig. 13, the 
details of video display processing by a video commu- 
nication device 115 according to this embodiment will 
be described. 

If video data is received in the step S502 shown in 
Fig. 1 2, the following the entire video display processing 
shown in Fig. 13 is performed. First, in a step S601, a 
data field 146 of received video data is read and digital 
video data is decompressed by video CODEC 113. 
Next, processing proceeds to a step S602 and in the 
step, a calling party's name field 142 is read to identify 
the calling party of video. Further, processing proceeds 
to a step S603 and in the step, it is checked whether 
video windows 910 to 960 on the display screen shown 
in Fig. 1 6 are empty or not. If they are empty processing 
proceeds to a step S604 and In the step, received video 
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data is displayed in a video windoVTcorresponding to a 
calling party of video. Concurrently, the time in which 
video Is fetched from the other field 45 is read and the 
time is displayed together with the name of a calling par- 
ty in any of statuses 91 1 to 961 . s 

If a plurality of calling parties of video exist in the 
network 1 30, received video data is displayed in a video 
window corresponding to the name of a catling party ob- 
tained in the step S602. 

In the meantime, if seven terminals or more of call- io 
ing parties of video exist in the network 130, video sig- 
nals for six terminals are already received and a video 
signal Is transmitted from the seventh terminal, in the 
step S603 processing is terminated without proceeding 
to the step S604 because data is stored in all the video is 
windows on the display screen shown in Fig. 16. 

In this embodiment, six video windows are provid- 
ed, however, seven video windows or more, or six video 
windows or less may be provided. In a word, the number 
of video windows may be changed depending upon that 20 
of calling parties of video in a network. 

In such a case, the number of calling parties of video 
concurrently displayed may be increased by creating a 
new screen with more video windows. 

By the above-described processing, video transmit- 
ted from the video communication devices 1 11 to 11 6 to 
the network 1 30 can be received and displayed at a real 
time. 

Next, referring to a flowchart shown in Fig. 14, the 
details of video retrieval processing by the video com- 30 
munication device 115 according to this embodiment will 
be described below. 

If a specific calling party of video is selected in a 
step S504 shown in Fig. 1 2, the following individual vid- 
eo display processing shown in Fig. 14 is performed. 55 

First, in a step S701 , it is checked whether an indi- 
vidual video display screen shown in Fig. 17 is generat- 
ed in a display apparatus 208 of the video communica- 
tion device 115 or not. If an individual video display 
screen is not generated, processing proceeds to a step 40 
S702 and in the step, an individual video display screen 
shown in Fig. 17 is generated. Next, in a step S703, it 
is checked whether the current mode button 1041 
shown In Fig. 17 Is selected or not. If the current mode 
button 1041 is selected, processing proceeds to a step ^5 
S704 and in the step, it Is checked whether an instruc- 
tion for operating a camera is Issued in the camera con- 
trol panels 1020 and 1030 or not. If an instruction for 
operating a camera Is issued, processing proceeds to a 
step S705 and in the step, control over a camera corre- so 
sponding to an instruction for operating a camera is ap- 
plied to a calling party of video. In the concrete, camera 
control information according to a format shown in Fig. 
1 8 is transmitted to a calling party of video, for instance 
video communication devices 111 to 116 shown in Fig. ss 
8. In the concrete, a code showing "control over a cam- 
era" is written to the packet type field 1 41 , the name of 
the local video communication device 115 is written to 
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the calling party's namWfeld 142, the name of a calling 
party obtained in the step S602 shown in Fig. 1 3 is writ- 
ten to a called party's name field 143, a code showing 
"control data" is written to the media type field 144, a 
camera control command is written to the other field 
145, and communication data to which additional infor- 
mation of the camera control command is written is gen- 
erated in the data field 1 46 and is transmitted to the net- 
work 1 30 via the communication control apparatus 1 29. 

Next, processing proceeds to a step S706 and in 
the step, magnified received video is displayed in a vid- 
eo window 1010 shown in Fig. 1 7 and processing is ter- 
minated. As the individual video window 1010 shown in 
Fig. 17 is larger than the entire video windows 910 to 
960 shown in Fig. 1 6, magnified video can be displayed. 

In the meantime, in a step S707, if it is detected that 
the past mode button 1042 shown in Fig, 17 is selected, 
processing proceeds to a step S708 and in the step, the 
following processing for retrieving and displaying the 
past image is performed. In the step S708, completion 
of the operation for setting the time at which video is 
input is awaited. When the time is input to the time input 
field 1051 In the time specification panel 1050 shown in 
Fig. 1 7 and it is detected that the setting button 1 052 is 
clicked, processing proceeds to a step S709 and in the 
step, the time input to the time input field 1051 is read. 

Next, processing proceeds to a step S710 and in 
the step, video data corresponding to the set time Is re- 
quired to the video storage apparatus 1 20 shown in Fig. 
8. In the concrete, a code showing "video data required" 
Is written to the packet type field 1 41 according to a for- 
mat shown in Fig. 18, the name of the local video com- 
munication device 11 5 is written to a calling party's name 
field 142, the name of the video storage apparatus 120 
is written to a called party's name field 1 43, a code show- 
ing "control data" is written to the media type field 144, 
and communication data to which the set time is written 
is generated in the other field 145 and is transmitted to 
the network 1 30 via the communication control appara- 
tus 119. 

The video storage apparatus 120 receives data ac- 
cording to a format shown in Fig. 8 and transmits re- 
quired video data. 

Next, processing proceeds to a step S711 and in 
the step, video data transmitted by the video storage ap- 
paratus 120 is received from the network 130 via the 
communication control apparatus 129 . In the next step 
S712, the data field 146 in received video data is read 
and digital video data is decompressed by video CO- 
DEC 113. Further, processing proceeds to a step S713 
and in the step, received video data is nnagnified In ad- 
justment with the size of the video window 1010 on the 
individual video display screen shown in Fig. 17 and is 
displayed in the video window 1010. 

Next, if displayed video is to be changed, the flow 
of processing shown In a flowchart in Fig. 1 2 is executed 
by turning on switches, flow shown In Fig. 14 is again 
executed and displayed video is changed. 
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By the above-described processing, the past video 
of video fetched from a remote video camera can be 
continuously retrieved based upon the current video and 
displayed. 

Next, referring to a flowchart shown in Fig. 15, the s 
details of video storage processing by the video storage 
apparatus 1 20 according to this embodiment will be de- 
scribed. 

First, in a step S801 , generation of an event is await- 
ed. If an event is generated, processing proceeds to a io 
step S802 and in the step, it is checked whether the type 
of the event is reception of video or not. If the packet 
type field 1 41 is checked and a code showing "transmis- 
sion of video" is identified when communication data 
shown in Fig. 18 is received from the network 130 via is 
the communication control apparatus 209 shown in Fig. 
1 0, processing proceeds to a step S803 and in the step, 
video data storage processing described below is per- 
formed. In the step S803, data in a calling party's name 
field 142 is read to identify the calling party of video. 20 

Next, data in the other field 145 is read to obtain 
time information comprising video. Video data written to 
the data field 146 is read and the video data is stored In 
an area in the secondary memory 206 corresponding to 
the calling party of the video together with the time in- 2S 
formation. If the name stored in a called party's name 
field 1 43 does not show the local video storage appara- 
tus, the received data is discarded. 

In the meantime, In the step S802, if the type of an 
event is not reception of video, processing proceeds to 30 
a step S804 and in the step, it is checked whether video 
data is required by video communication devices 111 to 
116 or not. If communication data including a code 
showing "video data required" is received from the net- 
work 1 30 via the communication control apparatus 209 3S 
shown in Fig. 10 and is written to the packet type field 
141, processing proceeds to a step S805 and in the 
step, the name of a requiring video communication de- 
vice written to a calling party's name field 1 42 and time 
information set in the other field 1 45 are obtained. Next, 40 
processing proceeds to a step S806, In the step, data Is 
retrieved using the name of the requiring video commu- 
nication device and the set time information as a key 
and desired video data is read from the secondary mem- 
ory 206 shown in Fig. 10. Next, processing proceeds to 45 
a step S807 and in the step, after a communication data 
format described later is generated, It is transmitted to 
the network 1 30 via the communication control appara- 
tus 209. 

In this case, the communication data complies with so 
a format shown in Fig. 1 1 , a code showing "transmission 
of video" Is written to the packet type field 1 41 , the name 
of the video storage apparatus 1 20 is written to a calling 
party's name field 142, the name of the requiring video 
communication device Is written to a called party's name ss 
field 143, a code showing "a dynamic image" is written 
to the media type field 144, time information Is written 
to the other field 1 45 and video data is written to the data 
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field 146. 

By the above-described processing, the past imag- 
es required by video communication devices 111 to 116 
can be transmitted to the network 1 30. 

The present invention may be applied to a system 
constituted by a plurality of devices or may be applied 
to a system comprising one device. The number of video 
communication devices is not limited to six. 

Also, naturally, the present invention can be applied 
to a system or a device on which a program is running. 

According to this embodiment as described above, 
as a system or an information communication device to 
which a storage and a plurality of information communi- 
cation devices are connected via a network Is provided 
with a transmission means for transmitting request data 
for requiring either Image data of an image input to an- 
other different from the above-described information 
communication device or an image at an arbitrary time 
stored in the above-described storage, and a receiving 
means for receiving transmission data transmitted in re- 
sponse to the above-described request data by the 
above-described another information communication 
device or the above-described storage, the current vid- 
eo provided by another information communication de- 
vice and an Image formerly obtained can be continuous- 
ly retrieved and displayed on the system. 



Claims 

1. An image processing apparatus comprising: 

first means for supplying an input image to first 
display means to display the Input Image at a 
real time; and 

second means for supplying an image prior by 
a predetermined time of said Input image to 
second display means to display the image at 
a real time. 

2. An image processing apparatus according to Claim 
1 , wherein said first display means and said second 
display means are respectively separate windows. 

3. An image processing apparatus according to Claim 
1 , further comprising storage means for storing said 
input image at a real time. 

4. An image processing apparatus according to Claim 
3, wherein said storage means stores said Input im- 
age in a file divided every a predetermined time. 

5. An image processing apparatus according to Claim 
1 , further comprising supply means for supplying a 
signal corresponding to said input image. 

6. An image processing apparatus according to Claim 
5, wherein said supply means is a telecamera. 
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7. An image processing apparaiSsaccordlng to Claim 
1 . further comprising setting means for setting said 
predetermined time. 

8. An image processing apparatus according to Claim 

7, wherein third display means for displaying said 
predetermined time set by said setting means is 
provided. 

9. An image processing apparatus according to Claim 

8, wherein said third display means is arranged on 
the same screen as said first display means and 
said second display means. 

10. An image processing apparatus for reading data 
from a file in which dynamic image data is stored, 
comprising: 

first reading means for reading said dynamic 
image data from said file at first timing; and 
second reading means for reading predeter- 
mined data in said dynamic image data at sec- 
ond timing different from said first timing in par- 
allel with said first reading means. 

11. An image processing apparatus according to Claim 
10, further comprising display means for displaying 
dynamic image data read by said first reading 
means and said second reading means. 

12. An image processing apparatus according to Claim 
10, further comprising said file in which said dynam- 
ic image data is stored. 

1 3. An image processing apparatus according to Claim 
10, wherein said predetermined data is static image 
data. 

14. An image processing apparatus according to Claim 
10, wherein said predetermined data is data prior 
by a predetermined time of dynamic image data 
read by said first reading means. 

15. An image processing apparatus according to Claim 
1 4, further comprising setting means for setting said 
predetermined time. 

16. An image processing apparatus for reading data 
from a file in which dynamic image data is stored, 
comprising: 

input means for inputting data required for se- 
lecting a dynamic Image to be read from dy- 
namic image data stored in said file; and 
control means for displaying specific plural im- 
ages prior to and later than said data based up- 
on data input by said input means on a monitor. 
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17. An image processV^apparatus according to Claim 
16, wherein said specific image is a static Image. 

1 8. An image processing apparatus according to Claim 
16, further comprising a setting means for setting 
said time. 

19. An image processing apparatus according to Claim 
16, further comprising said monitor. 

20. A communication device concurrently connected to 
a plurality of terminals, comprising: 

first selecting means for selecting one of said 
plural terminals concurrently connected; 
second selecting means for selecting either of 
dynamic image data stored in the selected ter- 
minal or image data input to said selected ter- 
minal; and 

reproduction means for reproducing either of 
dynamic image data selected by said selecting 
means or said input image data as a visible im- 
age. 



25 21. A communication device according to Claim 20, 
wherein said first selectlng-means selects a prede- 
termined terminal of twelve display means on which 
image data Input from said plural terminals concur- 
rently connected is concurrently displayed. 
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22. A communication device according to Claim 20, 
wherein said second selecting means selects said 
dynarriic image data by selecting the time at which 
said dynamic image data is stored. 

23. A communication device according to Claim 20, fur- 
ther comprising receiving means for receiving im- 
age data transmitted from the terminal selected by 
said first selecting means. 

24. A communication device according to Claim 23, 
wherein information showing a calling party is add- 
ed to said transmitted image data. 



45 25. A video communication system comprises a video 
communication device for transmitting and receiv- 
ing video data via a network and a video storage 
apparatus for storing video data received from said 
network, wherein: 
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said video communication device includes vid- 
eo input means; video transmission means for 
transmitting video data input by said video input 
means to said network; video receiving means 
for receiving video data from said network; 
means for reading video data stored in said vid- 
eo storage apparatus; means for receiving vid- 
eo data transmitted from said video storage ap- 
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paratus; means for seielnhg video data re- 
ceived by said video receiving means and video 
data received from said video storage appara- 
tus; and means for displaying dynamic image 
data selected by said selecting means; and s 
said video storage apparatus includes video re- 
ceiving means for receiving video data from 
said network; video storage means for storing 
video data received from said video receiving 
means; and a means for transmitting required io 
video data to said video communication device 
in case a request for reading is made from said 
video communication device. 

26. A video communication system according to Claim ^5 
25, further comprising: 

identification means for identifying a calling 
party of video data received from said network; 



display means for displaying a video signal is- 
sued by said video communication device 
based upon identification by said identification 
means. 

25 

27, A video communication system in which a current 
scene and a prior recorded scene can be displayed 
simultaneously on any one of a plurality of inter- 
linked display devices. 



and 



20 



30 



35 



40 



45 



50 



55 



11 



EP 0 715 466 A2 




FIG. 1 



20 



22 



CPU 



* — » 



MEMORY 



24 



HARD DISK 



28 



MOUSE 



< — » 



30 



CAMERA CONTROL 



32 



VIDEO CAPTURE 



VIDEO CAMERA 



BIT MAP DISPLAY 



r 



26 



14-1 



— — — J 




.14-2 

-L_ ig_2 



12 









^ 


VIDEO 

SERVER 
— J 



16-3 



14-3 



12 



EP 0 715 466 A2 



FIG. 2 



MAIN 
WINDOW 







TIME 




Ts 





tp 



tp 



TO 



FILE1 



Tb 



MAIN 
WINDOW 



Ts 



SUBWINOOW 



TO ^Jb 



FILE1 



FILE 2 



tp 



Ts 



SUBWINDOW 



TIME 



MAIN 
WINDOW 



TIME 



13 



EP 0715 466 A2 



FIG. 3 



( VIDEO READING PROCESSING ^ 



SET Ts 



-SI 



t = PRESENT TIME 



DISPLAY PRESENT VIDEO 
ON MAIN WINDOW 



Lr 



S2 
S3 




S5 



DISPLAY VIDEO OF 
tp ON SUBWINDOW 
, 



14 



EP 0 715 466 A2 



# 



FIO. 4 



< 



TO 



FILE 1 



tp 



Tb 



MAIN 
WINDOW 



TO 



FILE 2 



Ts 



Ts 



Tb 



tf 



tc 



SUBWINDOW 



SUBWINDOW 



FILE 3 



TO 



FILE1 



I FILE 2 



tp 



Tb 



MAIN 
WINDOW 



t' 

FILE 3 



Ts 



Ts 



tf 



tc 



SUBWINDOW 



SUBWINDOW 



15 



EP 0 715 466 A2 




Fia 5 



PAST VIDEO READING 
PROCESSING 



SET tc AND Ts 



t* = tc 



S11 



SI 2 



tf = tc + Ts 
tp = tc - Ts 



S13 





DISPLAY FRAMES OF tf 
AND tp ON SUBWINDOW 




r 


1 » 




DISPLAY VIDEO OF 

TIME t' ON MAIN WINDOW 



S14 



S15 




( END ^ 



16 



EP 0 715 466 A2 




FIG. 6 



DESK TOP VIDEO CONFERENCE 



88 



TIME 



"TFia .1 qftlAI DISPLAY . 
^^M •LiiiEl INTERVAL 



OSg MINUTE- 




15:25 15:30 15:35 

rni ( ppf 




^3 



Z 



YZ 



^4 







Wide 


Tele 








86 


84 



<1 



A 



V 



17 



EP 0 715 466 A2 




FIG. 7 
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FIG. 11 
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FIG. 12 
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